lets also add a check here that used memory is below maxmemory * 0.9 and rss curr memory is below
maxmemory * rss_oom_deny_ratio * 0.9

To make this test supper stable and more controlled
I would run
CONFIG SET enable_heartbeat_eviction false
before creating the connections
after that check the rss make sure its above the threshold
then run
CONFIG SET enable_heartbeat_eviction true
and after that check that rss memory dropped below the threshold and check the stats for evicted keys as you did below

incorrect results if max_memory > used_memory because size_t - size_t = size_t

Ok I got, the result is correct because you use overflow of ssize_t type, but it can be UB I'm not sure. please rewrite

Do you mean to round size_t to max ssize_t? Because I already implemented this previously
#4142 (comment)

just update ssize_t CalculateMemoryBudget(ssize_t max_memory, ssize_t used_memory)

But we are still converting size_t to ssize_t before calling the function. This is even more dangerous because we did not perform the subtraction.

Initially, we are using this values

atomic_uint64_t used_mem_peak(0);
atomic_uint64_t used_mem_current(0);
atomic_uint64_t rss_mem_current(0);
atomic_uint64_t rss_mem_peak(0);

in our case, such converting is absolutely safe, because we will never get such a memory size in the near future